Summary

Add OpenTelemetry tracing to backbeat so async work (replication, lifecycle,
GC, notifications) can be traced back to the original S3 request in Jaeger.
Arsenal already stamps value.traceContext.{traceparent,tracestate} onto every
MongoDB metadata write; the oplog carries it; this PR extracts it and
propagates it across backbeat's Kafka pipeline.

Gated by ENABLE_OTEL=true — when unset, init() returns before loading any
@opentelemetry/* package (zero overhead off the OTEL path).

Design and structure mirror the cloudserver OTEL PR
(scality/cloudserver#6140,
CLDSRV-884), which went through full human review.

Commits

Reviewable in order — each loads on its own and builds on the previous:

1. chore: add OpenTelemetry dependencies — explicit @opentelemetry/* packages, no auto-instrumentations-node bundle.
2. feat: add OTEL trust-boundary host filter — lib/tracing/trustedHosts.js (operator-supplied OTEL_TRUSTED_HOSTS).
3. feat: add OTEL SDK bootstrap and tracing facade — lib/tracing/index.js (init/close/isEnabled), sampler, span limits, traces-only, outbound-only HTTP, mongodb/ioredis/aws-sdk instr, bounded shutdown flush.
4. feat: propagate trace context across the Kafka pipeline — kafkaTraceContext.js, producer 7th-arg headers, publish() headers param, consumer span (links, never parent-child).
5. feat: instrument replication and oplog-populator pods.
6. feat: instrument lifecycle, GC, and notification pods.

What it does

1. lib/tracing/ facade — public surface is init() / close() /
   isEnabled(); OTEL internals are hidden. init() fails fast (assert) if
   ENABLE_OTEL=true and OTEL_EXPORTER_OTLP_TRACES_ENDPOINT is unset, and on
   an out-of-range OTEL_SAMPLING_RATIO. No baked-in endpoint default.
   Traces-only NodeSDK (logRecordProcessors:[] / metricReaders:[]),
   explicit span limits, ParentBasedSampler({ root: TraceIdRatioBased(ratio) })
   (default 1%) so a sampled upstream trace is always honored.

2. Explicit instrumentations — instrumentation-http +
   instrumentation-aws-sdk + instrumentation-ioredis (with
   requireParentSpan) + instrumentation-mongodb (with
   enhancedDatabaseReporting:false for PII masking). No
   auto-instrumentations-node bundle (avoids ~36 unused instrumentations,
   version skew, and a transitive @types/pg conflict). MongoDB is
   instrumented because several pods use the mongodb driver directly —
   oplog-populator (MongoLogReader tailing the oplog), lifecycle conductor
   (scan-path collection queries), and notification (MongoConfigManager).

3. HTTP: outbound only — instrumentation-http runs with
   disableIncomingRequestInstrumentation: true. No instrumented pod serves
   application HTTP — they are Kafka producers (oplog-populator, conductor) and
   consumers (replication, lifecycle, GC, notification processors); the only
   inbound HTTP is k8s probes / Prometheus scrapes, never a useful trace entry.
   So server spans are never created, which also removes the need to maintain a
   health-path ignore list. (The backbeat API server in bin/backbeat.js has
   real routes and is not instrumented here; instrumenting it later must
   re-enable server spans.)

4. Trust boundary — lib/tracing/trustedHosts.js. Trusted hosts come from
   an operator-supplied OTEL_TRUSTED_HOSTS env var (comma-joined lowercase
   bare hostnames); loopback is always trusted. Outbound calls to untrusted
   hosts (replication destinations, AWS/Azure/GCP, remote Artesca) have
   traceparent/tracestate stripped and the client span tagged
   scality.trace.suppressed. Unset env → loopback-only (safe default).
   Emitting the var for backbeat pods is an operator follow-up (cf. cloudserver's
   ZKOP-551).

5. Kafka propagation = span Links — every consumer read starts a new
   trace (via ROOT_CONTEXT) and adds an OTEL Link to the upstream span (never
   parent-child). Async work fires minutes/hours after the S3 request; links
   keep each trace small and navigable via Jaeger's link UI instead of
   producing million-span, multi-hour waterfalls. Producer/consumer header
   plumbing rides node-rdkafka's MessageHeader[] (array of single-key
   objects). Consumer spans set the OTEL messaging semantic-convention
   attributes (messaging.system, messaging.destination.name,
   messaging.destination.partition.id, messaging.consumer.group.name) and
   are marked ERROR on task failure.

6. Graceful flush — each pod's SIGTERM path calls tracing.close(), which
   is race-safe and bounded at 5s (Promise.race + .unref()'d timer) so an
   unreachable collector can't block past Kubernetes' 30s grace period.

Pods instrumented

oplog-populator, replication (data + status), lifecycle (conductor, bucket,
object/transition), GC, notification. Entry points call
require('../lib/tracing').init() before any HTTP/aws-sdk/ioredis/mongodb
module loads.

Compatibility note

Rebased onto development/9.4 (node-rdkafka ^2.12.0 → ^3.6.0). Verified the
produce() 7th-arg headers signature and MessageHeader[] format are
unchanged in v3.

Tests

Unit tests under tests/unit/lib/tracing/ (index boot/close/isEnabled +
fail-fast asserts, trustedHosts env parsing + request-hook strip/IPv6,
kafkaTraceContext) plus notification populator trace-header propagation.

Status

Reworked to mirror the reviewed design of cloudserver #6140; earlier automated
review comments triaged and resolved; split into 6 reviewable commits. Cluster
end-to-end verification with Jaeger + OTEL collector is the remaining open item.

Behavioral note: replicationStatusProcessor's SIGTERM handler now
force-exits with process.exit(0) after the trace flush. The original had no
success-path exit (it relied on the event loop draining naturally); this aligns
it with the other seven pods and makes shutdown deterministic. Intentional —
flagged explicitly so reviewers don't mistake it for an accidental change.

Issue: BB-764